[SLES-2971] feat(traces): gate X-Ray-driven sampling behind DD_MERGE_XRAY_TRACES - #1337
Closed
lym953 wants to merge 1 commit into
Closed
Conversation
|
lym953
force-pushed
the
yiming.luo/sles-2971-xray-sampled-priority
branch
from
August 25, 2026 03:11
e93c41c to
a17b8d8
Compare
…XRAY_TRACES Adopting an AWS-generated X-Ray `Sampled` flag as a Datadog sampling decision is opt-in behavior in the tracer libraries, where `DD_MERGE_XRAY_TRACES` defaults to false. bottlecap never read it, so there was no way to decline. Add the config and gate the flag on it. A header a Datadog library planted is never gated -- its `Sampled` is a Datadog decision regardless of the setting, and gating it would break dd-trace-java's SQS propagation, which uses this same header and has nothing to do with X-Ray. Scope is the sampling verdict only. Trace and parent IDs are still taken from an AWS-generated header either way, so correlation is unchanged by this flag; that is pre-existing v88+ behavior and gating it is a separate argument. The flag is read off `DatadogCompositePropagator`, which already carries the config to every extraction site -- threading `Arc<Config>` through the listener, LWA proxy, and interceptor state instead would have touched five more files.
lym953
force-pushed
the
yiming.luo/sles-2971-merge-xray-gate
branch
from
August 25, 2026 03:13
9e2f4db to
006b1f7
Compare
Contributor
Author
|
Closing: the gate isn't the right shape. Branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1325 — review that first; this diff is only the gate.
Problem
After #1325, an AWS-generated
Sampled=1still becomes an explicit Datadog priority of1. X-Ray's sampling decision therefore still drives Datadog's, and a user who wants their own sampling to govern has no way to decline.The tracer libraries make this opt-in through
DD_MERGE_XRAY_TRACES(DD_MERGE_DATADOG_XRAY_TRACESin Ruby), which defaults to false. bottlecap has never read it.Background
For an SQS trigger, the extension falls back to the X-Ray
AWSTraceHeaderwhen the record carries no_datadogattribute. Any Lambda→SQS→Lambda chain has such a header, because the AWS SDKs plant one for recursion detection, so this fallback fires for pipelines that have nothing to do with X-Ray.Change
Adds a
merge_xray_tracesconfig (DD_MERGE_XRAY_TRACES, default false) and gates the flag on it:Sampled=11— keep1— keepSampled=00— drop0— dropSampled=11— keep1— keepSampled=00— drop"Datadog-planted" means the root id carries the
…-00000000<16 hex>padding a Datadog library writes; anything else is AWS-generated. Only the last row moves with the flag.Default false, so behavior holds at #1325's until a user opts in.
Datadog-planted headers stay ungated
dd-trace-javaplants context in this same header for SQS, and it is the only carrier for JMS→SQS. Gating that on an X-Ray flag would break Datadog-to-Datadog propagation for users with no X-Ray involvement.Correlation stays ungated
Trace and parent ids come from an AWS-generated header in both states. That is how bottlecap has always behaved, it costs no spans, and gating it would leave the reporting customer with their spans restored but their producer→consumer link gone.
Turning merging on reinstates the drop
With merging on, an AWS-generated
Sampled=0drops the trace, which is the reported behavior — restored deliberately for anyone who asks for it, since the flag means X-Ray's decisions count.Where the flag is read
Off the propagator, which already carries the config to every extraction site. Threading the config explicitly would touch five more files, including two request-state tuples.
Naming
This reuses
DD_MERGE_XRAY_TRACESrather than introducing a new variable, so users find the setting where they already expect it. That broadens what the name covers: the docs currently scope trace merging to Node.js and Python and describe it as merging X-Ray's own spans, so they need a line for this path.Testing
Verified live on the SLES-2971 repro (Go, arm64,
provided.al2023, SQS trigger), sending each header shape in both flag states. Every span in the trace carried the priority shown.Sampled=01— kept0— droppedSampled=11— kept1— keptSampled=00— dropped0— droppedSampled=11— kept1— keptOnly the AWS-generated
Sampled=0row moves with the flag, which is the intent. A trace id was present in all eight runs, so correlation is unaffected either way.🤖 Generated with Claude Code